Skip to content

Conversation

@iwasakims
Copy link
Member

https://issues.apache.org/jira/browse/BIGTOP-4084

This PR bumps the verison of Tez to 0.10.3. We can remove most of the patch needed for Tez 0.10.2.

  • TEZ-4492, TEZ-4493, TEZ-4520 are contained in Tez 0.10.3.
  • We do not need TEZ-4319 since snappy-java-1.1.8.2 (supporting aarch64) is pulled as transitive dependency of hadoop now.

@iwasakims
Copy link
Member Author

I got protobuf related error on smoke-tests of Tez. I'm looking into the cause.

java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto,

@iwasakims
Copy link
Member Author

While hadoop uses shaded protobuf now, Hadoop 3.3.6 retains dependency on protobuf-java-2.5.0 for compatibility and the protobuf-java-2.5.0.jar is in the classpath. Since TEZ-4363 upgraded the protobuf to 3.21., Tez must use its own protobuf-java-3.21.1.jar.

@iwasakims iwasakims changed the title BIGTOP-4084. Upgrade Tez to 0.10.3. BIGTOP-4084. Upgrade Tez to 0.10.4. Oct 17, 2024
@iwasakims
Copy link
Member Author

Since init-hcfs.groovy is contained in hadoop package, smoke-tests need hadoop-pkg with the patch applied. You can manually put the tez.tar.gz on HDFS otherwise.

$ ./docker-hadoop.sh --create 1 ... --stack hdfs,yarn,mapreduce,tez
$ ./docker-hadoop.sh --exec 1 sudo -u hdfs hdfs dfs -put /usr/lib/tez/lib/tez.tar.gz /apps/tez/lib/
$ ./docker-hadoop.sh --smoke-tests tez

@sekikn
Copy link
Contributor

sekikn commented Oct 26, 2024

+1, tested with Debian 11 and Rocky 9 on x86_64. Thanks @iwasakims.

@iwasakims iwasakims merged commit 2870a5c into apache:master Oct 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants